Building on Deep Learning

We propose using deep learning as the “workhorse” of a cognitive architecture. We show how deep learning can be leveraged to learn representations, such as a hierarchy of analogical schemas, from relational data. This approach to higher cognition drives some desiderata of deep learning, particularly modality independence and the ability to make top-down predictions. Finally, we consider the problem of how relational representations might be learned from sensor data that is not explicitly relational. Deep Learning as a Workhorse for Cognition We consider the hypothesis, suggested by neuroanatomy (Mountcastle 1978), that higher level cognition is built on the same fundamental building blocks as low-level perception. Likewise, we propose that learning high-level representations uses many of the same mechanisms as learning perceptual features from low-level sensors, which is essentially what deep learning systems do. In our work, we assume that such a system —a system that not only learns a feature hierarchy from a collection of fixed-width vectors, but also uses the feature hierarchy to parse new vectors and make predictions about missing values— can be used as the workhorse for learning and reasoning. We assume that such a system is modality independent and learns a feature hierarchy with relevant invariances for whatever modality it is trained on, given enough training data. For example, given a large number of images, the system should learn features such as visual objects with invariance to rotation, translation, and scale. A copy of the same initial (untrained) system, given ample speech data, should learn phonemes and words with invariance to pitch, speed, and speaker. Some evidence suggests that the perceptual cortex is capable of such plasticity (Sur and Rubenstein 2005). There are already deep learning systems that accomplish part of this goal (Le et al. 2012), (LeCun 2012), but these provide the architecture and connectivity, which implicitly relies on knowledge of the topology of the sensor modalities on which these systems are trained. Ideally, we would like this network structure to be learned because, for higher-level representations, such as that described in the next section, the topology is unknown beforehand and must be learned. Copyright c © 2013, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. Though there is still work to be done by the deep learning community before such a system is completely developed, we consider how this system might be leveraged to learn and use higher level representations. Leveraging Deep Learning for Relational Data and Logical Inference A criticism of deep learning, and connectionism in general, is that such systems are incapable of representing (much less learning) relational schemas such as “sibling”. Furthermore, deep learning has been criticized for being unable to make simple parameterized logical inferences such as “If A loves B and B loves C, then A is jealous of C.” (Marcus 1998). We have taken steps to address these criticisms by showing how a second (non-connectionist) system can transform relational data into fixed-width vectors such that overlap among these vectors corresponds to structural similarity in the relational data. Unlike related approaches ((Socher et al. 2012), (Rachkovskij, Kussul, and Baidyk 2012), (Levy and Gayler 2008)), our representation is able to exploit partial analogical schemas. That is, a partial overlap in our representation’s vectors corresponds to a common subgraph in the corresponding structures. Furthermore, through processes of windowing and aliasing our system is able to represent structures with hundreds of entities and relations using a few thousand features, whereas the earlier work requires thousands of features to represent structures with only a handful of entities and relations. The details of our transformer and the examples below are given in (Pickett and Aha 2013). With this transformer, we can feed transformed structures into a simple deep learning system to learn features that are relevant for these structures. These learned features correspond to analogical schemas. For example, given 126 stories in predicate form (Thagard et al. 1990), our system produces a feature hierarchy of stories (corresponding to plot devices), part of which is shown in Figure 1. In this figure we see a “Double Suicide” analogical schema found in both Romeo & Juliet and in Julius Caesar. In the former, Romeo thinks that Juliet is dead, which causes him to kill himself. Juliet, who is actually alive, finds that Romeo has died, which causes her to kill herself. Likewise, in Julius Caesar, Cassius kills himself after hearing of Titinius’s death. Titinius, who is actually alive, sees Cassius’s corpse, and kills himself. The 37 Learning Rich Representations from Low-Level Sensors: Papers from the AAAI 2013 Workshop